[Monitoring] Out of the box alerting #68805

chrisronline · 2020-06-10T18:45:31Z

Refactors #62793
Refactors #61685
Relates to #42960

This PR introduces quite a few things to the Stack Monitoring UI:

A new visual approach to alerts and how they are surfaced, configured, and acted upon
The first additional out-of-the-box alert for monitoring CPU usage across Elasticsearch
Integrating new out-of-the-box alerts with existing watcher-based cluster alerts (until Provide migration path for monitoring alerting from Watcher to Kibana alerting elasticsearch#50032 is resolved)

Creation

We want these alert to be created/enabled by default without the user needing to know or do anything, but that is not yet possible. As a temporary solution, we will attempt to create these alerts every time the monitoring UI is loaded. If the alerts already exist, nothing will happen (duplicate alerts will not be created).

Visibility

Firing scenario

The alerts will appear in the UI when triggered, such as:

Clicking into the alert itself results in a list view with all "firing" alerts and the timestamp of when the alert was triggered (this data is stored by the alerting team)

Clicking into a single alert will give the user some useful information about the details of the alert as well as potential resolution steps:

Clicking View alert configuration will present the familiar flyout:

Non firing scenario

So, these screenshots showcase the visibility of firing alerts, but users can also gain visibility into these alerts through setup mode.

The UX will be the same when clicking on alerts in this context, except there will not by any useful information about the alert as it's not firing, but the View alert configuration button will exist allowing the user to change properties of the alert.

Legacy watcher-based alerts

Unfortunately, we are not able to permanently disable existing watcher-based cluster alerts until elastic/elasticsearch#50032 is resolved.

In the meantime, we will allow them to co-exist with our new Kibana alerts. The alerts themselves will exist as Kibana alerts, but will require the watch history to indicate a firing scenario before the Kibana alert itself fires. We are doing this because we can't stop the watches from doing what they do now, which will index into .monitoring-alerts-* and send an email (if configured). We don't want to step

The user will not be able to know the difference, and once we can fully disable watcher-based cluster alerts, we can convert these to full-fledge Kibana alerts behind the scenes and the user doesn't need to know.

Testing

To enable these new alerts, all you need to do is pull down this PR, start Kibana, and navigate to the Stack Monitoring UI. This will create all alerts, but there is slightly more you'll need to do to actually test the majority of the alerts.

The CPU usage alert will just work, but you'll need to enable watcher for the rest to work which means you'll need to on the trial license (or gold+). After doing that, you can verify they exist by going to Stack Management -> Watches and they should show up there (but keep in mind this bug will require you to enable legacy monitoring for watches to exist).

The harder part is actually getting your cluster in a state to trigger the various alerts.

CPU Usage

To simulate this, edit the threshold by enabling setup mode on the cluster overview page, clicking on the Alerts badge on the Nodes panel and editing the cpu usage alert configuration to some low value you can easily reach on your machine.

License expiration

For this, I've been adding an ingest pipeline to simulate an early expiration.

See https://gist.github.com/chrisronline/9d4d3d740e535d3c01410cac2cc74653

Cluster status

For this, simply create an index and add a document. This should trigger the alert indicating you need to allocate replica shards.

ES nodes change

For this, I found it easier to simulate with an ingest pipeline

See https://gist.github.com/chrisronline/d441aba1a08cb45082e59f39cc9f6687

Elasticsearch version mismatch

This is not testable, as the watch itself is broken. See elastic/elasticsearch#58261

Kibana version mismatch

Again, I found this easier to simulate with an ingest pipeline.

See https://gist.github.com/chrisronline/34328d14738f0ce754e36ec7031e45a9

Logstash version mismatch

Again, I found this easier to simulate with an ingest pipeline.

See https://gist.github.com/chrisronline/3b982d95710ef820d11c7443a1e49091

TODO

…a model

chrisronline · 2020-07-13T19:27:58Z

cc @gchaps for help with the copy here

Let me know if it's easier if I just prepare the copy in a doc or something

gchaps · 2020-07-13T21:28:13Z

@chrisronline Are all the screenshots included in the summary? If so, I can go by that. Might be best to set up a meeting and go through the copy together.

chrisronline · 2020-07-14T01:43:28Z

@gchaps

Here is the copy we need help with. This might be a bit confusing, so happy to clarify if necessary

Cluster health alert

Params

{state} - The current state of the alert.'
{clusterHealth} - The health of the cluster
{clusterName} - The name of the cluster to which the nodes belong.
{action} - The recommended action to take based on this alert firing

EDIT:
{clusterHealth}: The health of the cluster. (add ending period)
{clusterName: The cluster to which the nodes belong.
{action}: The recommended action for this alert.

Possible param values

{state}

firing
resolved

{action}

Allocate missing primary and replica shards.

UI messaging

When firing

Elasticsearch cluster health is {clusterHealth}.

When resolved

Elasticsearch cluster health is green.

Email

Subject: Cluster health alert is {state} for {clusterName}. Current health is {clusterHealth}.
Body: {action}

EDIT:
Cluster health is {state} for {clusterName}. Current health is {clusterHealth}.

Server log

Cluster health alert is {state} for {clusterName}. Current health is {clusterHealth}. {action}

CPU usage alert

Params

{state} - The current state of the alert.'
{nodes} - The list of nodes that are reporting high cpu usage.
{count} - The number of nodes that are reporting high cpu usage.'
{clusterName} - The name of the cluster to which the nodes belong.'
{action} - The recommended action to take based on this alert firing

EDIT:

{nodes}: The list of nodes reporting high CPU usage.
{count}: The number of nodes reporting high CPU usage.
{clusterName}: The cluster to which the nodes belong.
-{action} - The recommended action for this alert. (be sure to add ending period).

Possible param values

{state}

firing
resolved

{action}

Investigate (links into Kibana)

UI messaging

When firing

The cpu usage on node {nodeName} is now under the threshold, currently reporting at {cpuUsage}% as of #resolved

EDIT
The CPU usage on node {nodeName} is under the threshold. Usage is {cpuUsage}% as of #resolved.

When resolved

Elasticsearch cluster health is green.

Email

Subject: CPU usage alert is {state} for {count} node(s) in cluster: {clusterName}
Body: {action}

EDIT
Subject: CPU usage is {state} for {count} node(s) in cluster {clusterName} (Can we remove the word alert? Also colon before clusterName.

Server log

CPU usage alert is {state} for {count} node(s) in cluster: {clusterName}

Elasticsearch version mismatch alert

Params

{state} - The current state of the alert.'
{clusterName} - The name of the cluster to which the nodes belong.'
{versionList} - The list of unique versions.

EDIT
{clusterName} - The cluster to which the nodes belong.'
{versionList} - The versions of Elasticsearch running in this cluster. (may not be right, but should define what versions are being talked about).

Possible param values

{state}

firing
resolved

UI messaging

When firing

There are different versions of Elasticsearch ({versions}) running in this cluster.

EDIT
Multiple versions of Elasticsearch ({versions}) are running in this cluster.`

When resolved

All versions are the same for Elasticsearch in this cluster.

EDIT
All versions of Elasticsearch are the same in this cluster.

Email

Subject: Elasticsearch version mismatch alert is {state} for {clusterName}
Body: Elasticsearch is running {versionList} in {clusterName}.

EDIT
Subject: Multiple versions of Elasticsearch running in {clusterName}. (If not ok to remove alert state, leave as is)

Server log

Elasticsearch version mismatch alert is {state} for {clusterName}

Kibana version mismatch alert

Params

{state} - The current state of the alert.'
{clusterName} - The name of the cluster to which the instances belong.'
{versionList} - The list of unique versions.

EDIT
{clusterName} - The cluster to which the Kibana instances belong. (Instances is vague by itself. Is Kibana right here?)
{versionList} - The versions of Kibana running in this cluster.

Possible param values

{state}

firing
resolved

UI messaging

When firing

There are different versions of Kibana ({versions}) running in this cluster.

EDIT
Multiple versions of Kibana ({versions}) are running in this cluster.

When resolved

All versions are the same for Kibana in this cluster.

EDIT
All versions of Kibana are the same for this cluster.

Email

Subject: Kibana version mismatch alert is {state} for {clusterName}
Body: Kibana running {versionList} in {clusterName}.

EDIT
Subject: `Multiple versions of Kibana running in {clusterName}

Server log

Kibana version mismatch alert is {state} for {clusterName}

Logstash version mismatch alert

Params

{state} - The current state of the alert.'
{clusterName} - The name of the cluster to which the nodes belong.'
{versionList} - The list of unique versions.

EDIT
{clusterName} - The cluster to which the nodes belong.
{versionList} - The versions of Logstash running in this cluster.

Possible param values

{state}

firing
resolved

UI messaging

When firing

There are different versions of Logstash ({versions}) running in this cluster.

EDIT
Multiple versions of Logstash ({versions}) are running in this cluster.

When resolved

All versions are the same for Logstash in this cluster.

EDIT
All versions of Logstash are the same for this cluster.

Email

Subject: Logstash version mismatch alert is {state} for {clusterName}
Body: Logstash running {versionList} in {clusterName}.

EDIT
Subject: Multiple versions of Logstash running in {clusterName}

Server log

Logstash version mismatch alert is {state} for {clusterName}

License expiration alert

Params

{state} - The current state of the alert.'
{expiredDate} - The date when the license expires.
{action} - The recommended action to take based on this alert firing.
{clusterName} - The name of the cluster to which the nodes belong.'

EDIT
{action} - The recommended action for this alert.
{clusterName} - The cluster to which the nodes belong.

Possible param values

{state}

firing
resolved

UI messaging

When firing

This cluster's license is going to expire in #relative at #absolute. #start_linkPlease update your license.#end_link

EDIT
The license for this cluster expires in #relative at #absolute. #start_linkPlease update your license.#end_link

When resolved

This cluster's license is active.

EDIT
The license for this cluster is active.

Email

Body: Your license will expire in {expiredDate} for {clusterName}. {action}

EDIT
Body: Your license for {clusterName} expires in {expiredDate}. {action}

Server log

License expiration alert is {state} for {clusterName}. Your license will expire in {expiredDate}. {action}

EDIT
License expiration alert is {state} for {clusterName}. Your license expires in {expiredDate}. {action}

ES nodes changed alert

Params

{state} - The current state of the alert.
{clusterName} - The name of the cluster to which the nodes belong.
{added} - The list of nodes added to the cluster.
{removed} - The list of nodes removed from the cluster.
{restarted} - The list of nodes restarted in the cluster.

EDIT
{clusterName} - The cluster to which the nodes belong.
{added} - The nodes added to the cluster.
{removed} - The nodes removed from the cluster.
{restarted} - The nodes restarted in the cluster.

Possible param values

{state}

firing
resolved

UI messaging

When firing

Note: These will come out side by side if more than one apply.
Elasticsearch nodes '{added}' added to this cluster.
Elasticsearch nodes '{removed}' removed from this cluster.
Elasticsearch nodes '{restarted}' restarted in this cluster.

When resolved

No changes detected in Elasticsearch nodes for this cluster.

EDIT
No changes in Elasticsearch nodes for this cluster.

Email

Subject: Elasticsearch nodes changed alert is {state} for {clusterName}
Body: The following Elasticsearch nodes in {clusterName} have been added:{added} removed:{removed} restarted:{restarted}

EDIT

Subject: Elasticsearch nodes changed for {clusterName}

Can you format the body as follows:

The following Elasticsearch nodes changed in {clusterName}:

Added:{added}
Removed:{removed}
Restarted:{restarted}

Server log

Elasticsearch nodes changed alert is {state} for {clusterName}

igoristic

The changes look good 👍

kibanamachine · 2020-07-14T21:49:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 9cb8f7e

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
monitoring	217	+15	202

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* First draft, not quite working but a good start * More working * Support configuring throttle * Get the other alerts working too * More * Separate into individual files * Menu support as well as better integration in existing UIs * Red borders! * New overview style, and renamed alert * more visual updates * Update cpu usage and improve settings configuration in UI * Convert cluster health and license expiration alert to use legacy data model * Remove most of the custom UI and use the flyout * Add the actual alerts * Remove more code * Fix formatting * Fix up some errors * Remove unnecessary code * Updates * add more links here * Fix up linkage * Added nodes changed alert * Most of the version mismatch working * Add kibana mismatch * UI tweaks * Add timestamp * Support actions in the enable api * Move this around * Better support for changing legacy alerts * Add missing files * Update alerts * Enable alerts whenever any page is visited in SM * Tweaks * Use more practical default * Remove the buggy renderer and ensure setup mode can show all alerts * Updates * Remove unnecessary code * Remove some dead code * Cleanup * Fix snapshot * Fixes * Fixes * Fix test * Add alerts to kibana and logstash listing pages * Fix test * Add disable/mute options * Tweaks * Fix linting * Fix i18n * Adding a couple tests * Fix localization * Use http * Ensure we properly handle when an alert is resolved * Fix tests * Hide legacy alerts if not the right license * Design tweaks * Fix tests * PR feedback * Moar tests * Fix i18n * Ensure we have a control over the messaging * Fix translations * Tweaks * More localization * Copy changes * Type # Conflicts: # x-pack/plugins/monitoring/common/constants.ts # x-pack/plugins/monitoring/public/components/cluster/overview/alerts_panel.js # x-pack/plugins/monitoring/public/components/cluster/overview/index.js # x-pack/plugins/monitoring/public/components/elasticsearch/node/node.js # x-pack/plugins/monitoring/public/components/elasticsearch/nodes/nodes.js # x-pack/plugins/monitoring/public/components/kibana/instances/instances.js # x-pack/plugins/monitoring/server/plugin.ts # x-pack/test/functional/apps/monitoring/cluster/alerts.js

chrisronline · 2020-07-15T00:14:14Z

Backport

7.x: 510a684

chrisronline added 30 commits May 14, 2020 16:01

First draft, not quite working but a good start

d4be8c1

Merge in master

ca6981e

More working

be3f757

Support configuring throttle

6ea56e8

Get the other alerts working too

2f2955b

More

9a460f4

Separate into individual files

7f7aa37

Menu support as well as better integration in existing UIs

6d6aa63

Red borders!

dc81674

Merge remote-tracking branch 'elastic/master' into monitoring/guardrails

f8dc391

New overview style, and renamed alert

cbd61ef

more visual updates

6a613a7

Merge in master

fe33433

Merge remote-tracking branch 'elastic/master' into monitoring/guardrails

4e5b25c

Update cpu usage and improve settings configuration in UI

6b991db

Merge remote-tracking branch 'elastic/master' into monitoring/guardrails

2dc61d1

Convert cluster health and license expiration alert to use legacy dat…

cecf0e9

…a model

Merge remote-tracking branch 'elastic/master' into monitoring/guardrails

7cdcbae

Remove most of the custom UI and use the flyout

0f2fa45

Add the actual alerts

8ed790a

Remove more code

5605eca

Merge in master

f8537c3

Merge remote-tracking branch 'elastic/master' into monitoring/guardrails

a353097

Fix formatting

3ed13fe

Merge remote-tracking branch 'elastic/master' into monitoring/guardrails

1bb2584

Fix up some errors

1eb1e38

Remove unnecessary code

34dd501

Updates

f04deb0

add more links here

66263a0

Fix up linkage

103cdd0

Moar tests

8235195

igoristic approved these changes Jul 14, 2020

View reviewed changes

chrisronline added 6 commits July 14, 2020 09:16

Fix i18n

0c32aca

Ensure we have a control over the messaging

371165f

Fix translations

7e1c835

Tweaks

7fca12a

More localization

43a8344

Copy changes

2dcf32c

elastic deleted a comment from kibanamachine Jul 14, 2020

Type

9cb8f7e

elastic deleted a comment from kibanamachine Jul 14, 2020

chrisronline merged commit 06b1820 into elastic:master Jul 14, 2020

chrisronline deleted the monitoring/guardrails branch July 14, 2020 22:00

chrisronline mentioned this pull request Jul 14, 2020

[7.x] [Monitoring] Out of the box alerting (#68805) #71780

Merged

chrisronline mentioned this pull request Jul 20, 2020

[Monitoring] Enable setup mode for cloud to configure alerts #72511

Closed

lcawl mentioned this pull request Jul 30, 2020

[DOCS] Add Kibana alerts to Stack Monitoring #73762

Merged

2 tasks

Zacqary mentioned this pull request Aug 19, 2020

CPU Usage alert condition page is blank #75414

Closed

chrisronline mentioned this pull request Aug 27, 2020

[Monitoring] License check for watcher is wrong #64095

Closed

chrisronline mentioned this pull request Nov 25, 2020

[Monitoring] Respect xpack.monitoring.show_license_expiration setting #84347

Closed

chrisronline mentioned this pull request Dec 14, 2020

[Stack Monitoring] [Test Scenario] Out of the box alerting #85841

Closed

23 tasks

chrisronline mentioned this pull request Jan 5, 2021

[Monitoring] Migrate data source for legacy alerts to monitoring data directly #87377

Merged

chrisronline mentioned this pull request Mar 1, 2021

[Stack Monitoring] [Test Scenario] Out of the box alerting #93072

Closed

24 tasks

simianhacker mentioned this pull request Apr 29, 2021

[Stack Monitoring] [Test Scenario] Out of the box alerting #98765

Closed

24 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Monitoring] Out of the box alerting #68805

[Monitoring] Out of the box alerting #68805

chrisronline commented Jun 10, 2020 •

edited

Loading

chrisronline commented Jul 13, 2020

gchaps commented Jul 13, 2020 •

edited

Loading

chrisronline commented Jul 14, 2020 •

edited by gchaps

Loading

igoristic left a comment

kibanamachine commented Jul 14, 2020

chrisronline commented Jul 15, 2020

[Monitoring] Out of the box alerting #68805

[Monitoring] Out of the box alerting #68805

Conversation

chrisronline commented Jun 10, 2020 • edited Loading

Creation

Visibility

Firing scenario

Non firing scenario

Legacy watcher-based alerts

Testing

CPU Usage

License expiration

Cluster status

ES nodes change

Elasticsearch version mismatch

Kibana version mismatch

Logstash version mismatch

TODO

chrisronline commented Jul 13, 2020

gchaps commented Jul 13, 2020 • edited Loading

chrisronline commented Jul 14, 2020 • edited by gchaps Loading

Cluster health alert

Params

Possible param values

{state}

{action}

UI messaging

When firing

When resolved

Email

Server log

CPU usage alert

Params

Possible param values

{state}

{action}

UI messaging

When firing

When resolved

Email

Server log

Elasticsearch version mismatch alert

Params

Possible param values

{state}

UI messaging

When firing

When resolved

Email

Server log

Kibana version mismatch alert

Params

Possible param values

{state}

UI messaging

When firing

When resolved

Email

Server log

Logstash version mismatch alert

Params

Possible param values

{state}

UI messaging

When firing

When resolved

Email

Server log

License expiration alert

Params

Possible param values

{state}

UI messaging

When firing

When resolved

Email

Server log

ES nodes changed alert

Params

Possible param values

chrisronline commented Jun 10, 2020 •

edited

Loading

gchaps commented Jul 13, 2020 •

edited

Loading

chrisronline commented Jul 14, 2020 •

edited by gchaps

Loading